Lukáš Polák
Lukáš Polák has been playing with distributed systems and ETL for several years. In the past, he helped Hollywood movie studios to ingest analytical data from social media platforms, so they could improve the aim of their ads. Also, he supported the design and creation of a proprietary query engine named Husky. The engine was designed to allow low-code access to data, even for non-technical users.
Currently, he works on a platform for data ingestion and calculation of insights for Data Stories, using project Prefect.
Scaling distributed workflow using Prefect Talk
English language
Lukáš PolákIn Ataccama, we looked for a way to deploy, manage, schedule and scale python-based workloads - data processing jobs or automated extraction of business intelligence. We are thrilled to share our experience with Prefect and Dask with you. This setup allowed us to seamlessly scale our workflows in Kubernetes environment. In this talk, we compare Prefect with other popular frameworks (ease of use, unit tests, observability, …), how we approach workload management and demonstrate dev-to-production path.